Papers with masked language modelling
Towards Zero-Shot Multimodal Machine Translation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Current multimodal machine translation systems rely on fully supervised data, which is costly to collect and prevents extension of MMT to language pairs with no such data. |
| Approach: | They propose a method to bypass the need for fully supervised data to train MMT systems . they adapt a strong text-only machine translation model to a visually conditioned language model and a divergence test set to evaluate how well models use images to disambiguate translations. |
| Outcome: | The proposed method can generalize to languages with no fully supervised training data. |
Compositional Zero-Shot Domain Transfer with Text-to-Text Models (2023.tacl-1)
Copied to clipboard
Fangyu Liu, Qianchu Liu, Shruthi Bannur, Fernando Pérez-García, Naoto Usuyama, Sheng Zhang, Tristan Naumann, Aditya Nori, Hoifung Poon, Javier Alvarez-Valle, Ozan Oktay, Stephanie L. Hyland
| Challenge: | Existing approaches to zero-shot domain transfer are limited by domain gap and lack of in-domain labels. |
| Approach: | They propose a compositional transfer learning framework (DoT51) that learns domain knowledge and task knowledge in a multi-task manner without access to in-domain labels. |
| Outcome: | The proposed framework outperforms the current state-of-the-art in zero-shot domain transfer by over 7 absolute points in accuracy on RadNLI. |
Word Order Matters When You Increase Masking (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that word order information is not always useful for pre-trained models to solve downstream tasks. |
| Approach: | They examine the effect of removing position encodings on the pre-training objective itself to see whether models can reconstruct position information from co-occurrences. |
| Outcome: | The proposed model can reconstruct position information from co-occurrences without explicit encoding, but not on some tasks. |
On Losses for Modern Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Devlin et al. ( 2018) released a transformer network (BERT) pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP). |
| Approach: | They clarify NSP's effect on BERT pre-training and explore ways to include multiple tasks into pre-train. |
| Outcome: | The proposed framework outperforms BERTBase on the GLUE benchmark using fewer than a quarter of training tokens. |
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained vision-and-language BERTs aim to learn representations that combine information from both modalities. |
| Approach: | They propose a diagnostic method based on cross-modal input ablation to assess the extent to which pretrained models integrate cross-module information. |
| Outcome: | The proposed method evaluates the model's performance on the other modality based on inputs from one or both modality. |
Sequence Reducible Holdout Loss for Language Model Pretraining (2024.lrec-main)
Copied to clipboard
| Challenge: | Data selection techniques have shown empirical benefits in reducing the number of gradient steps to train neural models. |
| Approach: | They propose to modify an existing data selection technique to adapt it to the sequence losses typical in language modeling. |
| Outcome: | The proposed technique reduces the number of steps required to train neural models by 4.3% and improves generalization ability on out of domain datasets. |